External Vision Systems (XVS) 

Proof-of- Concept Flight Test Evaluation 

Kevin J. Shelton , Steven P. Williams, Lynda J. Kramer, Jarvis (Trey) J. Arthur, 
Lawrence (Lance) Prinzel III, and Randall E. Bailey 
NASA Langley Research Center, Hampton, VA, USA 23681 


ABSTRACT 

NASA’s Fundamental Aeronautics Program, High Speed Project is performing research, development, test and 
evaluation of flight deck and related technologies to support future low-boom, supersonic configurations (without 
forward-facing windows) by use of an external Vision System (XVS). The challenge of XVS is to determine a 
combination of sensor and display technologies which can provide an equivalent level of safety and performance to that 
provided by forward-facing windows in today’s aircraft. This flight test was conducted with the goal of obtaining 
performance data on see-and-avoid and see-to-follow traffic using a proof-of-concept XVS design in actual flight 
conditions. Six data collection flights were flown in four traffic scenarios against two different sized participating traffic 
aircraft. This test utilized a 3x1 array of High Definition (HD) cameras, with a fixed forward field -of- view, mounted on 
NASA Langley’s UC-12 test aircraft. Test scenarios, with participating NASA aircraft serving as traffic, were presented 
to two evaluation pilots per flight - one using the proof-of-concept (POC) XVS and the other looking out the forward 
windows. The camera images were presented on the XVS display in the aft cabin with Head-Up Display (HUD)-like 
flight symbology overlaying the real-time imagery. The test generated XVS performance data, including comparisons to 
natural vision, and post-run subjective acceptability data were also collected. This paper discusses the flight test 
activities, its operational challenges, and summarizes the findings to date. 
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1. INTRODUCTION 

NASA’s High Speed Project in the Fundamental Aeronautics Program is addressing several research areas enabling the 
development of a low-boom/no-boom supersonic aircraft for flight over land. The new low-boom design requires a very 
long slender nose that prohibits effective forward-facing windows (Figure 1). The challenge then becomes the 
development of technologies which can provide an equivalent level of safety and performance to that provided by 
forward-facing windows in today’s aircraft. 



Figure 1. NASA F-15 Test Aircraft / Gulfstream Quiet Spike [1], and Supersonic Airline concept 


1.1 Background 

XVS is a combination of sensor and display technologies which may provide an equivalent level of safety and 
performance to that provided by forward-facing windows in today’s aircraft. Significant research was conducted under 
NASA’s High Speed Research (HSR) program during the 1990s on the design and development issues associated with 
an XVS for a conceptual High Speed Civil Transport (HSCT) aircraft [2]. What emerged from this work - which still 
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holds true today - is that the key challenge for an XVS design exists during VFR operations and when it is assumed that 
the flight crew has natural visibility (whether or not they may be operating on an Instrument Flight Rules (IFR) flight 
plan). Therefore, the driving XVS design standards emerge from the three tenets of Visual Flight Rules (VFR) 
operations which apply to all aircraft: “ see-and-avoid ”, “ see-to-follow ”, and “ self-navigation ” [3]. XVS is critical to the 
economic success of a supersonic aircraft with a nose boom, since the pilot’s lack of forward visibility would severely 
restrict aircraft operations and airspace usage especially when the weather is clear and visibility conditions are 
unrestricted - i.e., without an XVS, a low-boom supersonic aircraft cannot operate under Visual Flight Rules (VFR) 
since it would be unable “see-and-avoid” and “see-to-follow.” 

Previous work in this area investigated and verified the suitability of the display and vision system in basic aircraft 
operations including landing and low weather operations [4]. This flight test effort investigates issues with the 
integration of supersonic aircraft in the national airspace, for VFR/Visual Meteorological Conditions (VMC) operations, 
without need for special handling by air traffic control. 

In the Uninhabited Air Vehicle (UAV) sector, the “sense-and-avoid” technologies are actively being pursued and their 
work may be applicable [5]. These concepts are maturing and may be utilized in a support role towards meeting some of 
the XVS operational requirements, particularly the “see-and-avoid” requirement. 

1.2 Objectives 

The primary objective of the flight test was to safely obtain exploratory data on see-and-avoid and see-to-follow 
capability using a proof-of-concept XVS in real-world flight conditions. This work is expected to lead to further efforts 
exploring the extent of the technology’s performance in a larger data set of real-world conditions. 

2. METHODOLOGY 


2.1 XVS POC Flight Test 

NASA Langley Research Center’s Beechcraft UC-12 aircraft was selected as the evaluation aircraft (Figure 2). Installed 
on the inside of the right windscreen were three High Definition (HD) cameras serving as the XVS sensors with a fixed 
forward field-of-view (FOV). The camera images were displayed on a matching FOV XVS monitor with flight 
symbology overlaying the real-time imagery. Two evaluation pilots flew on each flight, with Subject 1 in the cockpit 
right seat looking Out-the- Window (OTW) and Subject 2 at the XVS display in the cabin (Figure 2). 



Figure 2. NASA UC-12 (NASA528) Test Aircraft and Cabin Layout 


This configuration provided a simultaneous evaluation of traffic aircraft detection, identification, and threat assessment 
while using XVS compared to natural vision OTW in the same operational conditions. The evaluation pilot looking out 
the window was, in a sense, a ‘control’ for XVS pilot in same operating conditions. The primary metric was the measure 
of time for traffic detection between the two evaluation subject crew; visually out-the- window and with the XVS 
display. Additionally, several runs were conducted with the safety pilot simulating ATC traffic callouts directing the 
evaluation pilot’s scan. Other evaluations were conducted such as the cameras’ seams, clutter, latency, resolution, and 
the use of traffic cuing symbology. Custom hardware and software were developed that enabled the monitoring and 
recording of the evaluation pilots event marker buttons as well as the XVS display de -clutter switch. The flight test crew 
included a NASA Safety Pilot who was the pilot in command at all times, two evaluation subjects, and one 
operator/researcher. Several demonstration flights included additional personnel as observers. 


2.2 Flight Test Operations 

Each data collection sortie consisted of multiple see-and-avoid and see-to-follow traffic scenarios. At mid-flight, the 
evaluation pilots swapped positions and roles. The evaluation pilots flew on two sorties so that each evaluation pilot 
observed two different size traffic (medium and small) aircraft at each observer location (OTW, XVS). Serving as the 
small traffic was the NASA Langley’s Cessna 206 and the Cirrus SR-22 shown in Figure 3. Acting as a medium- sized 
traffic aircraft was NASA Langley’s B200 King Air shown in Figure 4. 



Figure 3: NASA Langley's Cessna 206 (NASA 504) and Cirrus SR-22 (NASA 501), Small Traffic Aircraft 



Figure 4: NASA Langley’s B200 King Air, Medium Traffic Aircraft NASA 529 

2.3 Operating Location 

All test flights originated and terminated at NASA Langley (KLFI) airfield. The evaluation flight testing operations 
were conducted in south eastern Virginia in the vicinity of and at Wakefield Municipal Airport (KAKQ). The 
participating traffic aircraft departed before the evaluation aircraft and joined up at the test operating area. The see-and- 
avoid operations occurred over a long straight section of railway southeast of KAKQ (Figure 5). The see-to-follow 
operations were conducted at the nearby Wakefield airport. 



Figure 5. Operating Location 



2.4 XVS Proof-of-Concept Instrumentation 


A 3x1 array of Iconix Studio 2k HD cameras were arranged in portrait mode for a resulting FOV of 51° wide by 30° high 
with a pixel density of 63 pixels per deg. The minimal design objective of Snellen visual acuity was achieved and 
laboratory testing demonstrated the end-to-end system acuity was better than 20/20. Each of the cameras had a 
resolution of 1080x1920 for a total of 3240x1920 pixels. Each synchronized camera channel operates at 60 frames per 
second and was mixed (Figure 6) with computer generated symbology and displayed on the XVS display which 
consisted of 3 -tiled LCD panels. The LCD panel pixel density and arrangement mirrored the camera pixel density and 
FOV to create a conformal display. 
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Figure 6. XVS Hardware Block Diagram, Display Design Eye Reference 


The XVS display was mounted mid-cabin with a 25” design eye reference point to achieve the same field-of-view as the 
cameras. There was no overlap of the camera FOVs (Figure 7, right photo). 



Figure 7. XVS Cameras and Display Installation 


The small size of the Iconix camera heads enabled them to be mounted at the top of the windscreen on the right side. 
(Figure 7, left photo) 






Figure 8. XYS Triple Display 


The symbology of the XVS System (Figure 8) utilized in the flight test included typical HUD elements such as: horizon 
line with heading indications, pitch ladder with boresight reference, airspeed tape, altitude tape, and heading compass. A 
traffic designator box, using Automatic Dependent Surveillance-Broadcast (ADS-B), was also drawn based upon 
experimental condition. 

2.5 Evaluation Pilots 

Volunteers were recruited to serve as Evaluation Pilots (EP). The evaluation pilot pool included civil servants and 
Department of Defense pilots. Six EPs participated in the flight evaluations. Two were USAF and the rest were civilian 
and represented a mix of operational and flight experience. The EPs had an average of 1250 flying hours and 17 years of 
experience. All EPs had current medical certificates and a minimum of 20/20 corrected vision. 

2.6 Training 

All evaluation pilots were given a pre-flight briefing which covered the following topics: Experiment Background, 
Project plan, Test Objectives, Schedule, Operations Summary, XVS Operations, Test Conduct & Crew Procedures, and 
Question & Answer. The evaluation pilots were briefed on their task, which consisted of 4 parts: 1) detection of traffic, 
2) identification of traffic, 3) recognition if the traffic poses a hazard, and 4) avoidance. At the end of each data trial, 
pilots were given questionnaires regarding tasks and their workload. 

2.7 Evaluation Pilot Tasks/Actions: 

The evaluation pilot’s main task was to visually acquire traffic from their given position (OTW or the XVS). Once 
acquired, the pilot would press an event marker button and point out the traffic. The event marker button press is 
recorded with a time stamp in the data. 

2.8 XVS See-and-Avoid Procedures 

The scenarios were designed such that the pilot could only detect traffic via their given position. The scenarios were 
also designed so the traffic aircraft would be first-detected in the forward view quadrant, within the FOV of the XVS. 

At the beginning of the run, the evaluation pilots were instructed to begin the traffic scan task and press the event button 
when traffic was spotted. There were two different see-and-avoid scenarios selected from a number of possible 
scenarios utilized in previous flight tests. Two nose-to-nose scenarios were chosen as the most difficult in detecting 
traffic aircraft, as the traffic aircraft is simply expanding and not translating in the FOV [4]. 

ADS-B data was provided to the Safety Pilot for scenario set-up and safety-of-flight. A temporary cardboard vision 
restriction device was placed such that the OTW evaluation pilot could not see the ADS-B display. 


Scenario 1A - See-and-Avoid, Nose-to-nose, Horizontal Offset 

This nose-to-nose scenario had the evaluation aircraft and traffic aircraft in long closed loop patterns, with close but non- 
conflicting ground tracks shown in Figure 9. Both the evaluation and traffic aircraft were at the same altitude. The 
traffic aircraft approached from the southeast and maintained a path parallel and East of the railway tracks. 



Figure 9. Scenario 1 A - See-and-Avoid, Nose-to-Nose Horizontal Offset (not to scale) 

The evaluation aircraft approached from the Northwest and maintained a path parallel and West of the railway tracks. 
The lateral separation between the aircraft paths was 1000 feet (ft.) or less, but not less than 500 ft. Ground reference 
points (railway tracks) and electronic navigational displays with custom waypoints were used for coordination. Aircraft 
depicted in Figure 9 are in the start position. Note that Figure 9 is not to scale, thus the oncoming traffic appeared in the 
11 to 12 o’clock position, expanding and not translating in the field of view. 

Scenario IB - See-and-avoid, Vertical Offset, Traffic Below 

This nose-to-nose scenario had laterally overlapping ground tracks but with a vertical separation. The traffic aircraft, 
simulating a departure, was climbing towards the test aircraft and appears as a stationary expanding traffic in the ground 
clutter. The initial start configuration is shown in Figure 10, with the aircraft vertically separated by approximately 3500 
ft. The evaluation aircraft was in level flight at approximately 4500 ft. during the entire scenario. The traffic aircraft 
began to climb from approximately 1000 ft., simulating a departure and then leveled at an altitude 500 ft. below the 
evaluation aircraft. The aircraft climb rate was approximately 1000 ft. /min. 



Figure 10. Scenario IB - See- and- A void, Vertical Offset, Plan and Profile View (not to scale) 




2.9 XVS See-to-Follow Procedures 


There were two different see-to-follow scenarios selected for typical traffic interaction in the terminal area: an in-trail to 
a single runway and parallel runway scenarios. The EPs were instructed to scan for traffic and press the event button 
when the traffic was acquired or reacquired due to maneuvering. 

Scenario 2A - See-to-Follow, In-Trail; Following Traffic Aircraft 

This scenario had the evaluation aircraft follow the traffic aircraft around an extended visual landing pattern. 
Approaches were flown at a constant altitude, as landings were not required. The evaluation aircraft’s pattern altitude 
was 1500 ft AGL and the traffic aircraft was 1000 ft AGL. The evaluation aircraft followed as #2 in-trail at a minimum 
distance of Va to Vi statute miles, shown in Figure 11. 



Scenario 2B - See-to-Follow, In-Trail; Following Traffic Aircraft 

This scenario simulated simultaneous parallel runway approaches, with the traffic aircraft in the lead and approaching a 
landing to a designated ground reference acting as a virtual runway. As in the previous scenario, approaches were flown 
at a constant altitude, and the minimum slant distance between aircraft was Va to Vi statute miles, shown in Figure 3 at 
three different points of time (T1-T3). 



Figure 12. Scenario 2B - See-to-Follow, In-Trail, Plan View 






2.10 Post-Run 


After each run the evaluation pilots were given the following questionnaire to assess the run and assign a workload 
rating. Post-run question (1A) asked the pilots to rate the ease in detecting traffic within the visual scene marking the 
Likert scale (Figure 13) from 1 (Very Hard) to 7 (Very Easy). Similarly, post-run question (IB) asked the pilots to rate 
the ease in identifying traffic within the visual scene using the same scale. 


Very Somewhat Somewhat Very 

Hard Hard Hard Neutral Easy Easy Easy 


| 1 2 3 4 5 6 7 | 

Figure 13. Ease of Detecting and Identification Likert Scale 

Post-run question 2 asked the pilots to provide ratings for the statement ‘7 had sufficient time to assess and react to the 
traffic I detected ” on a Likert scale from 1 (Strongly Agree) to 7 (Strongly Disagree). Finally, the Air Force Flight Test 
Center (AFFTC) Pilot Workload Estimate was admitted and the evaluation pilots rated their workload on the following 
scale: 


1 - Nothing to Do; No System Demands 

2 - Light Activity; Minimum Demands 

3 - Moderate Activity; Easily Managed; Considerable Spare Time 

4 - Busy; Challenging But Manageable; Adequate Time Available 

5 - Very Busy; Demanding To Manage; Barely Enough Time 

6 - Extremely Busy; Very Difficult; Non-Essential Tasks Postponed 

7 - Overloaded; System Unmanageable; Important Tasks Undone 

2.11 Post-flight 

During the post-flight debriefing, unstructured free-form pilot comments were solicited and a post-test questionnaire was 
administered. The EPs were asked to provide an assessment of their display preference and their perceived safety (using 
a Likert scale) during operations for the tasks that they just completed flying. 

3. RESULTS 


3.1 Flight Summary 

Table 1 is a summary of the six XVS data collection flights. There were ten (10) flights in total including an instrument 
check flight and three demonstration flights. All flight time totaled approximately 20 hours with the data collection 
flights comprising approximately 12 hours. 

Table 1. Flight Summary 


Flight No 

EP# 

EP# 

Traffic Aircraft 

Estimated Visibility 
Conditions Up and Away 

1 

1 

2 

Small 

5 mi Haze 

2 

4 

3 

Small 

4 mi Haze 

3 

4 

3 

Medium 

15 mi Haze 

4 

1 

2 

Medium 

~10 mi Haze 

5 

5 

6 

Small 

7 mi Haze 

6 

5 

6 

Medium 

<5 mi Haze 


3.2 XVS Performance 

The ability of the EPs to visually acquire the traffic is summarized in Table 2. These data show the following trends: 





• When using the XVS during see-to-follow operations, the evaluation pilots were able to detect the traffic 
aircraft 100% of the time, regardless if it was a small or medium-sized aircraft (Table 2). This result was also 
true for evaluation pilots detecting traffic aircraft OTW. 

• In the co-altitude see-and-avoid runs (Scenario 1A), the EPs detected the traffic aircraft 17 of the 19 trials 
(90%) OTW. 

• Using the XVS display in this same scenario, EPs were only able to detect 8 of the 19 (42%) traffic aircraft 
(Table 2). Evaluation pilots were able to detect the conflicting traffic 33% and 50% of the time in the XVS 
display for the small and medium sized aircraft, respectively. 

• Traffic aircraft detection was better with the XVS for the climbing from below (Scenario IB) see-and-avoid 
runs with a detection rate of 63% and 88%. The OTW traffic detection rate was also better and was 88% for the 
small aircraft and 100% for the medium aircraft for the climbing from below see-and-avoid runs. 


Table 2. Traffic Detection Summary 


Scenario 

Number 

Scenario 

Type 

Scenario Description 

Size of 
Traffic 

Number of 
Aircraft 
Detected 

No. of 
Samples 

Percentage of 
Aircraft 
Detected 

XYS 

OTW 

XVS 

OTW 

1A 

S2A 

Traffic Nose-to-Nose, Co-Altitude, 
500'-1000' lateral offset 

Small 

3 

8 

9 

33 

89 

Medium 

5 

9 

10 

50 

90 

IB 

S2A 

Traffic Nose-to-Nose, Climbing from 
below, 500' low level off 

Small 

5 

7 

8 

63 

88 

Medium 

7 

8 

8 

88 

100 

2A 

S2F 

Following Traffic in pattern 
(No. 2 to land) 

Small 

6 

6 

6 

100 

100 

Medium 

4 

3 

4 

100 

75 

2B 

S2F 

Parallel runway operations, following 
traffic on parallel runway 

Small 

6 

6 

6 

100 

100 

Medium 

6 

6 

6 

100 

100 


3.3 Quantitative Analysis 

Though the sample size was not large, a statistical analysis was conducted on the quantitative data for the see-and-avoid 
scenarios. A Multivariate Analysis of Variance (MANOVA) was conducted on dependent measure of range-at-detection 
for the independent variables of traffic size (small, medium); scenario (Scenario 1A and IB); and subject observer 
position (XVS, OTW). If the pilot failed to detect the event, a value of ‘O’ was assigned. 

The MANOVA (Pillai’s Trace) was significant at p < 0.05 alpha for traffic size, F(3, 62) = 4.926; scenario, F(3,62) = 
10.831; and pilot observer, F(3,62) = 6.536. Subsequent Analysis of Variance (AN OVA) was conducted on the 
dependent measure for these significant MANOVA results. 

Traffic Size 

ANOVAs were conducted on the dependent measures of event detection range to traffic, relative altitude at time of 
detection, and time -to -closure at time of detection. A significant result was found for range, F(l, 64) = 13.189, p = 
0.001; and for time-to-closure, F(l,64) = 13.192, p = 0.001. Overall, pilots detected the medium-sized traffic (mean = 

1.3 nm) at greater distances than the small-sized traffic (mean = 0.9 nm) and this resulted in greater time-to-closure or 
collision when the traffic size was medium (17.1 sec) compared to the small traffic size (7.6 sec). 

Scenario 

The results of the ANOVA statistics evince a significant main effect for scenario for relative altitude only, F(l,64) = 
30.299, p = 0.0001. However, the result is due to the differences in scenarios (i.e., climbing compared to level flight) 
and serves to confirm that the scenarios achieved the proper geometries defined for the scenarios. 




Closer examination of the data purport that the above results for “traffic size” (i.e., medium was detected sooner and at 
greater range) was due largely to the scenario. This was confirmed when the data was isolated by scenario; specifically, 
a significant main effect for traffic size for the climbing scenario only (IB), F(l, 32) = 9.152, p= 0.005. 

Observer Position 

Subject pilots were assigned roles to detect traffic either (a) OTW; or, (b) with the XVS display. Significant main 
effects were found between OTW and XVS for range, F(l,64) = 17.144, p = 0.0001; relative altitude, F(l,64) = 4.799, p 
= 0.032; and time-to-closure, F(l,64) = 17.249, p = 0.0001. For range-to-traffic, the OTW pilots (mean = 1.4 nm) 
detected traffic at greater range than the XVS pilots (mean = 0.54 nm). The result for range-to-traffic was confirmed 
with significant results for OTW relative altitude at time of detection (-394 ft) compared to XVS (-227 ft); and further, 
the OTW observer station (17.9 sec) had greater time-to-closure compared to the XVS pilots (6.8 seconds). 

Air Traffic Control (ATC) Call-Outs 

The study further examined the effect of simulating ATC calling out traffic on OTW and XVS detection. The 
MANOVA was significant on these correlated dependent measures, F(3, 66) = 6.291, p = 0.001 based on Pillai’s Trace. 
Subsequent ANOVAs confirm the hypothesis that ATC traffic calls significantly enhanced pilot detection for range, 
F(l,68) = 11.980, p = 0.001; relative altitude, F(l,68) = 12.241, p = 0.001; and time-to-closure, F(l,68) = 8.859, p = 
0.001. On average, when provided simulated ATC traffic call outs, pilots detected the traffic at greater range (mean = 
1.6 nm) than without traffic calls (mean = 0.8 nm) which provide better awareness of traffic position based on relative 
altitude for the climbing scenario (-535 ft) than without ATC calls (-212 ft). The greater range and better traffic 
positional awareness resulting from the ATC traffic call-outs provide significantly more time to avoid the traffic (19.1 
sec) than without (10.2 sec). 

3.4 Qualitative analysis 

Workload Ratings by Scenario Number and Traffic Aircraft Size 

The evaluation pilot workload ratings are shown in Table 3. Overall, for all the flight trials conducted, the pilots rated 
workload as being a bit more demanding when using the XVS display (mean rating = 3.3) for detecting traffic aircraft 
than when looking OTW (mean rating = 2.9). Regardless if they were using the XVS display or OTW, the evaluation 
pilots’ workloads were easily managed and they had considerable spare time to attend to other tasks (see Table 3 and 
Figure 14). The ANOVA results found no main effects for observer position (OTW, XVS) on workload ratings, F(l,93) 
= 1.395, p = 0.241. 


Table 3. Workload 


Scenario 

Type 

No 

Scenario Description 

Size of 
Traffic 

XVS Workload 

OTW Workload 

Sample 

Size 

Mean 

Std 

Dev 

Sample 

Size 

Mean 

Std 

Dev 

See-and- 

avoid 

1A 

Traffic Nose-to-Nose, Co- 
Altitude, 500'-1000' lateral 

Small 

7 

3.6 

1.6 

9 

3.1 


Med 

10 

3.6 

1.5 

10 

3.0 


IB 

Traffic Nose-to-Nose, Climbing 
from below, 500' low level off 

Small 

7 

3.3 

1.4 

7 

3.1 


Med 

6 

4.2 

1.2 

8 

3.1 


See-to-follow 

2A 

Following Traffic in pattern 
(No. 2 to land) 

Small 

5 

3.0 

1.2 

4 

2.3 

0.5 

Med 

4 

3.0 

0.8 

3 

2.3 

0.6 

2B 

Parallel runway operations, 
following traffic on parallel Rwy 

Small 

6 

2.7 

0.8 

6 

3.2 

1.3 

Med 

6 

2.3 

0.5 

6 

2.5 

0.5 

Overall 

51 



53 

2.9 

0.9 











Frequency of Ratings 



1 - NothingTo Do; 

No System Demands 

2 - Light Activity; 

Minimum Demands 

3 - Moderate Activity; Easily Managed; 

Considerable Spare Time 

4 - Busy; Challenging But Manageable; 

Adequate Time Available 

5 - Very Busy; DemandingTo Manage; 

Barely Enough Time 

6 - Extremely Busy; Very Difficult; 

No n- Ess e ntia I Ta s ks P o stpo n e d 

7 - Overloaded; System Unmanageable; 

Important Tasks Undone 
No Rating 


Figure 14. Histogram of AFFTC Workload Ratings for All XVS POC Flight Test Trials 

Post-run questionnaires 

The subjective data from the EPs provided the same assessment that the quantitative data analysis revealed; a general 
lack of overall XVS performance. Regardless of scenario type (see-and-avoid or see-to-follow) or traffic aircraft size 
(small or medium), pilot ratings indicate that traffic aircraft detection (Question 1A) was easier OTW than in the XVS 
display. Overall ratings indicate that traffic aircraft detection was approaching somewhat easy to accomplish OTW 
(mean rating=4.4) and was somewhat hard to do with the XVS display (mean rating=3.0). The safety protocol’s 
repeated use of railroad tracks as a ground reference may have influenced the pilot’s ability to pick up traffic after 
several runs (i.e., they may have learned to look in the right general area), however both the XVS and OTW pilots would 
have been cued to this equally. 

Similarly, on Question IB, regardless of scenario type (see-and-avoid or see-to-follow) or traffic aircraft size (small or 
medium), pilot ratings indicate that traffic aircraft identification was easier OTW than in the XVS display. Overall 
ratings indicate that traffic aircraft identification was somewhat easy to accomplish OTW (mean rating=5.1) and 
somewhat hard with the XVS display (mean rating=3.3) (Figure 15). 



Ease Detecting 


Ease Identifying 


Figure 15. Boxplot of Evaluation Pilot’s Ease in Detecting and Identifying Traffic 

MANOVA and ANOVA statistical analyses were conducted on the post-run questionnaire data that comprised three 
questions for the independent variables of traffic size (small, medium) and observer position. The MANOVAs (Pillai's 
Trace) were significant for traffic size, F(4, 90) = 2.672; and observer position, F(4,90) = 4.013. The ANOVA analyses 
on these variables were conducted for the dependent measures of Question 1A, IB, and 2. The results evince a 
significant main effect for traffic size but only for Question IB, F(l,93) = 4.769, p = 0.049; pilots rated the medium 
sized aircraft to be easier to detect (Figure 16). For observer position, significant main effects were found for all three 
dependent measures: Question 1A, F(l,93) = 5.132, p = 0.026; Question IB, F(l, 93) = 12.901, p = 0.001; and Question 
2, F(l,93) = 4.830, p = 0.030. 
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Figure 16. Boxplot of Ease of Detecting and Identifying vs. Size 


On the aircraft assessment and reaction to it (Question 2), regardless of scenario type (see-and-avoid or see-to-follow) or 
traffic aircraft size (small or medium), evaluation pilots reported it was easier to accomplish OTW than in the XVS 
display. Overall ratings indicate that pilots agreed the OTW view (mean rating=2.3) provided them sufficient time to 


assess and react to traffic; however, this result was not found for the XVS display (mean rating=3.9) where pilots ratings 
indicated a neutral response for traffic assessment and reaction time. This matches the findings in the quantitative 
analysis. 

3.5 Discussion 

The test results generally proved the method to test for equivalent levels of performance between OTW and XVS for 
traffic detection and identification. The scenarios were extremely demanding, especially for the see-and-avoid 
maneuvers, emphasizing expansion with translation for the traffic. The nose-to-nose, see-and-avoid scenarios were 
chosen to represent a worst case, and test results confirmed the scenario to be the most difficult to detect for both the 
OTW and XVS subjects. 

XVS development to date has been focused towards increasing the pixel density to improve the visual acuity as it is a 
primary issue in the ability to detect and identify traffic. These data show that the XVS system as flown did not provide 
equivalent levels of performance. The flight test highlighted that secondary factors can be equally, if not more so, critical 
factors. 

For instance, while encountering bright hazy conditions, the commercial-off-the-shelf camera’s internally controlled 
parameters were unable to optimize the entire scene and the image contrast was insufficient to detect traffic at the range 
the OTW pilot detected traffic. The FOV of each camera, configured in portrait orientation, encompasses half sky and 
half ground. With bright hazy conditions, the camera adjusts its internal parameters but was unable to compensate for 
the very high dynamic range of the scene. The importance of dynamic range and dynamic contrast compensation cannot 
be minimized. Installation and positioning of the cameras is also critical. 

In the nose-to-nose, see-and-avoid scenarios - where the traffic is expanding without translation - the test results 
highlighted the performance differences between the OTW and XVS subjects. During the see-to-follow scenarios, the 
traffic is maneuvering and translating in the FOV which increased the ability to detect and identify. This aspect was 
enough to overcome the shortcomings of the camera in those scenarios (2 A, 2B). 

3.6 Issues & Lessons Learned 

The nature of POC testing is to discover issues. This initial exploratory flight test proved fruitful as many valuable 
issues were exposed in flight conditions. We found the operation of an array of sensors is feasible, but with an increased 
cost in system complexity. There was a significant effort and diligence required in the balancing and alignment of the 
cameras. The issues and lessons learned were numerous and are summarized below. 

• High end production grade cameras provide many controls but require much (operator) attention to optimize. 

• Bright haze is a difficult condition for the cameras systems to cope with. 

• Motion sickness due to system latency (and an un-collimated display) is a design issue to overcome. 

• Despite efforts, cockpit reflections in the cockpit were picked up by the cameras looking through windscreen. 

• The array of cameras worked but required necessary diligence in alignment and color balancing. 

• Maintaining precise color balance across cameras is an issue. 

• Circularly polarized filters were very helpful and increased contrast significantly. 

• Heat in the cockpit on the ramp subtly damaged a camera which reduced the image quality. 

• Heater wires in the windscreen were invisible under many conditions; however under some conditions they 
posed a large detriment to the image quality. 

• Bug splats will be an issue and will need to be overcome (possibly with redundancy, cleaner, wipers. . .). 

This test was the first round in the development spiral. Future designs will move toward custom camera options which 
create high pixel density, very high dynamic range, locally-adjustable exposure and contrast in image subsections, and 
minimal latency. 


4. SUMMARY 

An External Vision System Proof-of-Concept was flown in six data collection flights accomplishing 60 traffic encounter 
trials while evaluating two tenants of VFR operation; “see-and-avoid” traffic and “see-to-follow” traffic. This concept 
incorporated an array of three HD cameras and displays to increase the effective pixel density in the forward field of 
view. Laboratory testing demonstrated the system visual acuity was better than 20/20, however flight testing in actual 



conditions exposed deficiencies. The Iconix camera performed well in a previous flight test under near perfect visibility 
conditions. With increased pixel density, expectations were that equivalent visual capability might have been 
achievable. The results from the see-to-follow portion of the test suggest that this XVS system is equivalent to visual 
operations. However, in the see-and-avoid scenario, the XVS performance was lacking and far underperformed the 
visual OTW evaluation pilots. The concept XVS system did not achieve an equivalent visual capability and results 
demonstrate the system was inadequate for the tasks in the conditions tested. 
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